How to setup a network between two machines via RJ11 Modular Telephone Cable?

If your machines' RJ11 ports have analog telephone modems behind them (as opposed to being some kind of integrated HomePNA adaptors or something), then you could connect the telephone cord between the two and should be able to make the two modems connect to each other by using a comm port terminal program to issue an "ATD" (dial) command on one and an "ATA" (answer) command on the other. Assuming they were both 33.6K or later modems, and assuming they didn't freak out at the lack of dial tone or ring signals, they should handshake at 33.6kbps bi-directional. Later modems that were advertised as "56K" could only get 53-56kbps downstream, when connected to a digital modem connected to a digital phone circuit at the other end of the line. Analog telephone modems topped out at 33.6kbps when hooked up to analog phone lines like most people had in their homes.

Some modems may be able to handle just having a passive phone cord between them, but normally there would be telephone company equipment in between, providing up to 20mA of power at 24V (if I remember correctly). If your modems can't handle the passive phone cord, you may need to go to Radio Shack or your favorite electronics supply store and pick up a few components to create a simple "line simulator" to provide that DC voltage on the line. See here: http://www.jagshouse.com/modem.html

After getting the two modems to connect, you could then use the kermit/xmodem/ymodem/zmodem binary file transfer features in your comm port terminal programs to transfer files.

If you wanted it to be more of a real network connection and send IP packets across, you'd need to configure the PPP software in your OS to deal with the modems. You'd need to configure a PPP server on one end, and a PPP client on the other (depending on your OS and PPP software, that could be the same software package that could act in either role). Most PPP software probably assumes your modems are connect to a real telephone network that provides dial tone and ring signals, so you might have to fiddle with your PPP software to get it to configure your modems to not wait for a dial tone before dialing (this is known as "blind dialing"), and to not wait for a ring signal before answering.

If you're thinking 33.6 kbps is terribly slow for a clean pair of voice-grade telephone wires over just a few feet of cord between machines, you're right. The analog telephone system was baseband, which means that the sounds of your voice were translated into analog electromagnetic waves at the same frequencies as your voice (300 Hz - 3.4 kHz), so the phone system didn't guarantee that any higher frequencies would work across the telephone network. In fact the telephone companies determined that 8-bit, 8kHz sampling (64kbps data rate) was sufficient at reproducing intelligible human speech, so they standardized on that. But when hooked up to an analog line without any way to coordinate with the ADC (analog to digital converter) at the phone company, it's hard to make full use of that 64kbps data rate, so 33.6 was about all anyone could figure out a way to manage. In contrast, modern DSL modems leave the "baseband" 0-4kHz frequencies for analog voice service, and use the huge range of frequencies above that (this is called "broadband") to send data on. There are some practical limits to which frequencies broadband modems can use over voice grade telephone wire that may run, unshielded, for 5km before reaching the telephone company's central office, so that's why ADSL2+ tops out around 25mbps instead of reaching hundreds of megabits or gigabits per second.

Additional resources: The venerable Linux modem HOWTO